1. Field of the Invention
The present invention relates to an image processing technology of a dye-stained living specimen, and specifically relates to an image processing apparatus, an image processing method, and an image processing program for forming a display image from image data obtained by imaging a pathological specimen with multiband using transmitting illumination.
2. Description of the Related Art
In the living specimen, especially in the pathological specimen, magnifying observation with a microscope for acquiring various findings is widely performed after thinly slicing a block specimen obtained by removing an organ and the specimen obtained by performing a needle biopsy to approximately several micrometers. Among them, transmitting observation with an optical microscope is one of the most popular observation methods, because equipments are relatively inexpensive and easy to handle, and this has been performed through the ages. In this case, the thinly sliced living specimen hardly absorbs or scatters light and is nearly clear and colorless, so that this is generally stained by dye before the observation.
Conventionally, various types of staining methods have been suggested, and a total number thereof rises to 100 or larger; however, regarding especially the pathological specimen, hematoxylin-eosin staining (hereinafter, referred to as “H&E staining”) using blue-purple hematoxylin and red eosin as dyes is normally used.
Hematoxylin is a natural substance collected from a plant, and has no staining properties in itself. However, hematin, which is oxide thereof, is a basophilic dye, and combines with a negatively charged substance. Deoxyribo nucleic acid (DNA) in a cell nucleus is negatively charged by a phosphate group contained as a component, so that this combines with hematin and is stained in blue-purple. Meanwhile, as described above, although hematin, which is the oxide of hematoxylin, has the staining properties, this is hereinafter referred to as hematoxylin because this is general as the name of dye.
Eosin is an acidophilic dye and combines with a positively charged substance. A pH environment affects the charge of amino acid and protein, and they tend to be positively charged under an acidic condition. Therefore, eosin solution is sometimes used with acetic acid added thereto. The protein in a cytoplasm combines with eosin and is stained to red to pink.
In the specimen after the H&E staining, the cell nucleus, bone tissue, and the like are stained in blue-purple, and the cytoplasm, connective tissue, and a red blood cell are stained in red, respectively, and are easily visually recognized. As a result, an observer may comprehend sizes of components composing the tissue such as the cell nucleus and a positional relationship among them, so that it becomes possible to morphologically judge a state of the specimen.
The staining of the living specimen is an operation to fix the dye to living tissue originally having individual difference using chemical reaction, so that it is difficult to obtain an always uniform result. Specifically, even when the specimen is allowed to react with staining solution of the same concentration for the same time period, an amount of the dye to be fixed is not always nearly equal. According to the specimens, there are cases in which relatively more dyes are fixed and in which relatively less dyes are fixed. In the former case, the specimen is stained deeper than the general one; on the other hand, in the latter case, the specimen is stained lighter than the general one. There are facilities staffed with a stain engineer having specialized skills in order to prevent such variation in staining. In such facilities, the staining variation in the same facilities may be reduced to a certain degree by a professional adjustment operation by the stain engineer, but it is not possible to reduce the staining variation of other facilities.
The above-described staining variation has two problems. First, when a human visually observes, irregularity in states of the observation objects might cause stress of the observer. Especially, when there is significant variation, possibility of overlooking crucial finding cannot be denied.
Second, when imaging the stained specimen with a camera and image-processing the same, the staining variation might badly affect process accuracy. For example, even when it is known that a certain lesion displays a specific color, it is difficult to automatically extract this fact from the image. This is because the staining variation disturbs the color variation due to the lesion.
A method for solving such a problem in the staining variation by the image processing using a multiband image is disclosed in Tokiya Abe et al., “Color Correction of Pathological Images Based on Dye Amount Quantification”, OPTICAL REVIEW Vol. 12, No. 4 (2005) 293-300 (hereinafter, referred to as a non-patent document). In the non-patent document, a relative amount of the dye (dye amount) fixed from the multiband image to the stained specimen is estimated based on a predetermined physical model. Next, the estimated dye amount is virtually increased or decreased, and further a color image of the virtual specimen is synthesized using the increased or decreased dye amount. By appropriately increasing and decreasing the dye amount, the deeply stained specimen and the lightly stained specimen can be corrected to the image having the color nearly equal to the appropriately stained specimen. Hereinafter, the correction technique of the image disclosed in the above-described non-patent document will be described in more detail.
First, the multiband image is imaged with a frame sequential method by rotating 16 band path filters with a filter wheel to switch. Such an imaging method is disclosed in, for example, Japanese Patent Application Laid-Open No. 07-120324. With this imaging method, the multiband image having a 16-band pixel value at each point of the specimen is obtained.
The dyes are originally three-dimensionally distributed in the stained specimen, which is the observation object, but it is not possible to directly acquire the same as a three-dimensional image by a general transmitting observation system, and this is observed as a two-dimensional image obtained by projecting illuminating light passing through the specimen on an imaging device of the camera. Therefore, the above-described each point of the specimen means the point on the projected specimen corresponding to each pixel of the imaging device.
A relationship represented by the following equation (1) is established among a position (x, y) of the image, a pixel value g(x, y, b) in a band b, and a spectral transmittance t(x, y, λ) of a corresponding point on the specimen.g(x,y,b)=∫f(b, λ)s(λ)e(λ)t(x,y,λ)dλ+n(b)  (1)where λ represents a wavelength, f(b, λ) represents a spectral transmittance of a b-th filter, s(λ) represents spectral sensitivity characteristics of the camera, e(λ) represents spectral radiance characteristics of the illumination, and n(b) represents imaging noise in the band b. Meanwhile, b is a sequential number for identifying the band, and is an integral value satisfying 1≦b≦16.
In an actual calculation, the following equation (2), which is obtained by discretizing the equation (1) in a wavelength direction, is used.G(x,y)=FSET(x,y)+N  (2)Here, when a sampling number in the wavelength direction and the number of bands are set to D and B (wherein B=16), respectively, G(x, y) is a B-row 1-column matrix corresponding to the pixel value g(x, y, b) at the position (x, y). Similarly, T(x, y) is a D-row 1-column matrix corresponding to t(x, y, λ), and F is a B-row D-column matrix corresponding to f(b, λ). S is a D-row D-column diagonal matrix and a diagonal component corresponds to s(λ). E also is a D-row D-column diagonal matrix and the diagonal component corresponds to e(λ). N is a B-row 1-column matrix corresponding to n(b). In the equation (2), since the equations regarding a plurality of bands are summarized using the matrix, a variable b representing a band is not explicitly expressed. Also, integration of the wavelength λ is substituted by a matrix product.
Next, the spectral transmittance at each point of the specimen is estimated from the imaged multiband image. Wiener estimation is used as the estimation method at that time.
The Wiener estimation is widely known as one of linear filter methods for estimating an original signal from a signal on which noise is superimposed, and an estimation value {circumflex over (T)}(x,y) of the spectral transmittance can be calculated by the following equation (3).{circumflex over (T)}(x,y)=RSS(FSE)t((FSE)RSS(FSE)t+RNN)−1G(x,y)  (3)where RSS is a D-row D-column matrix and represents an autocorrelation matrix of the spectral transmittance of the specimen of the object specimen. RNN is a B-row B-column matrix and represents the autocorrelation matrix of noise of the camera used for imaging. Also, ( )t and ( )−1 represent a transposed matrix and an inverse matrix, respectively. Hereinafter, {circumflex over (T)} is sometimes represented as T^ (this notation applies to a case of a letter other than T).
Next, the dye amounts at each point (x, y) of the specimen are estimated based on the estimated spectral transmittance T^(x, y). Here, the dyes, which are objects of the estimation, are hematoxylin, eosin which stains the cytoplasm, and eosin which stains the red blood cell. Hereinafter, for simplicity, the above-described three kinds of dyes are abbreviated as a dye H, a dye E, and a dye R in this order. Meanwhile, to be strict, the red blood cell has its particular color even in a state without staining, and after the H&E staining, the color of the red blood cell itself and the color of eosin changed in the course of staining are superimposingly observed. Therefore, to be precise, the color obtained by superimposing the color of the red blood cell itself on the color of eosin, which stains the red blood cell, is referred to as the dye R.
Generally, it is known that in a light transmissive substance, a Lambert-Beer law represented by the following equation (4) is established between intensity I0(λ) of incident light and intensity I(λ) of emitting light for each wavelength λ.
                                          I            ⁡                          (              λ              )                                                          I              0                        ⁡                          (              λ              )                                      =                  exp          ⁡                      (                                          -                                  k                  ⁡                                      (                    λ                    )                                                              ·              d                        )                                              (        4        )            where k(λ) and d represent a substance-specific value decided depending on the wavelength λ and thickness of the substance, respectively. Also, the left side of the equation (4) represents the spectral transmittance.
When the H&E stained specimen is stained by three kinds of dyes H, E and R, the following equation (5) is established in each wavelength λ by the Lambert-Beer law.
                                          I            ⁡                          (              λ              )                                                          I              0                        ⁡                          (              λ              )                                      =                  exp          ⁡                      (                          -                              (                                                                                                    k                        H                                            ⁡                                              (                        λ                        )                                                              ·                                          d                      H                                                        +                                                                                    k                        E                                            ⁡                                              (                        λ                        )                                                              ·                                          d                      E                                                        +                                                                                    k                        R                                            ⁡                                              (                        λ                        )                                                              ·                                          d                      R                                                                      )                                      )                                              (        5        )            where kH(λ), kE(λ) and kR(λ) represent k(λ) corresponding to the dyes H, E and R, respectively. Also, dH, dE and dR represent virtual thicknesses corresponding to the dyes H, E and R, respectively. Originally, the dyes are dispersively present in the specimen, so that a concept of width is not correct; however, this becomes an index to indicate a relative dye amount indicating the relative amount of the dye which is present compared to a case supposing that the specimen is stained by a single dye. That is to say, it can be said that dH, dE and dR represent the dye amounts of the dyes H, E and R, respectively. kH(λ), kE(λ) and kR(λ) can be easily obtained from the equation (4) by creating the specimen stained by a single dye in advance and measuring the spectral transmittance thereof by a spectrometer.
The following equation (6) is obtained by taking a natural logarithm of both sides of the equation (5).
                                          -            log                    ⁢                                    I              ⁡                              (                λ                )                                                                    I                0                            ⁡                              (                λ                )                                                    =                                                            k                H                            ⁡                              (                λ                )                                      ·                          d              H                                +                                                    k                E                            ⁡                              (                λ                )                                      ·                          d              E                                +                                                    k                R                            ⁡                              (                λ                )                                      ·                          d              R                                                          (        6        )            The component corresponding to the wavelength λ of the estimation value T^(x, y) of the spectral transmittance at the position (x, y) on the specimen obtained by using the equation (3) is represented as t^(x, y, λ), and by substituting this t^(x, y, λ) into the equation (6), the following equation (7) is obtained.−log {circumflex over (t)}(x,y,λ)=kH(λ)·dH+kE(λ)·dE+kR(λ)·dR  (7)In this equation (7), unknown variables are three, which are dH, dE and dR, so that it is possible to make the equation (7) simultaneous with at least three different wavelengths λ to solve. In order to further improve accuracy, it is possible to make the equation (7) simultaneous with four or more different wavelengths λ to perform a multiple regression analysis.
Once the dye amounts dH, dE and dR are obtained, it is possible to simulate the variation in the dye amounts in the specimen by correcting them. That is to say, by multiplying appropriate coefficients αH, αE and αR, by dH, dE and dR, respectively, and substituting them into the equation (5), an equation (8) is obtained as a new spectral transmittance t*(x, y, λ).t*(x,y,λ)=exp(−(kH(λ)·αHdH+kE(λ)·αEdE+kR(λ)·αRdR))  (8)By substituting the equation (8) into the equation (1), the image of the specimen of which dye amount is virtually changed may be synthesized. In this case, however, it is possible to calculate by setting the noise n(b) to 0.
By the above-described procedure, it becomes possible to virtually adjust the dye amounts of the stained specimen. By preparing an appropriate user interface, a user himself may adjust the dye amount and confirm the result by the image. Therefore, even when there is staining variation in the stained specimen, the user may adjust the same to an appropriate staining state to observe, and the problem in the staining variation may be solved.